UKCLE guide to subject review: appendices

• preparatory activities for subject review
• aide-memoire for subject review – see Annex E of the QAA handbook
• reviewing student work
• observation of teaching and learning
• role of the facilitator – see Annex F of the QAA handbook
• agenda for meeting with students – see Annex J of the QAA handbook
• making judgements in subject review

Preparatory activities for subject review

There is a broad range of activities that you may wish to consider in preparing for, during and after subject review visits, including:

• briefings on the QAA subject review methodology
• sessions on preparing the self-evaluation (focusing on for example aims and learning outcomes, addressing standards, dealing with the different aspects of quality of learning opportunities)
• provision of briefing documentation including key QAA documents
• advice sessions from current QAA subject reviewers
• critical analysis of draft self-evaluation (by colleagues external to the department/school)
• sessions on dealing with benchmark statements on standards, the code of practice, course specifications and the national qualifications framework
• liaison with central service providers
• ‘dry runs’
• preparing for observation of teaching

Reviewing student work

(This information is based on the student work and assessment pro forma used in the methodology to the end of 2001 – it is likely however that the pro forma used in the new method will cover similar areas)

The student work and assessment note reflects the emphasis in the process on student work. In preparing for the visit the department/school will wish to ensure that the samples of student work to be provided will enable reviewers to make meaningful (and positive) comments in the areas covered by the note – these are described below.

The student work and assessment note invites reviewers to comment on strengths and weaknesses of students’ achievements in relation to the intended learning outcomes in the following areas:

• evidence of student preparation
• knowledge and understanding
• cognitive skills
• key skills (eg communication, numeracy, IT)
• subject-specific skills (including practical and professional skills)
• values, motivation and attitudes

Reviewers are further asked to comment on the strengths and weaknesses of assessment in relation to the intended learning objectives in the following areas:

• clarity of assessment task
• match of assessment method to learning outcomes
• appropriateness to level
• match to student profile
• appropriateness and clarity of assessment criteria
• consistency of application of marking criteria
• consistency of marking
• quality of feedback provided to students
• evidence of internal moderation

In addition, reviewers will also note the overall quality of students’ work, assessment employed and any relevant contributions by external examiners.

Note that when providing examples of student work departments/schools should ensure that marking and feedback sheets, and assessment criteria, accompany the samples. Where oral feedback has been given to students in addition to any written feedback then this should be clearly indicated.

It is the responsibility of departments/schools to check thoroughly the accuracy of marking and the consistency and quality of the feedback provided on the sample of work.

Observation of teaching and learning

The observation note

(This information is based on the observation pro forma used in the methodology to the end of 2001 – it is likely however that the pro forma used in the new method will cover similar areas. It should be read in conjunction with Annex I of the QAA handbook.)

The observation note requires reviewers to report on the specific learning objectives planned for the session and to comment on strengths and weaknesses in relation to the learning outcomes in the following areas:

• clarity of intended learning outcomes
• planning and organisation
• methods/approach
• delivery and pace
• content (currency, accuracy, relevance, use of examples, level, match to student needs)
• student participation
• use of accommodation and learning resources

In addition, reviewers are asked to summarise the session’s overall quality in relation to the learning outcomes.

Making judgements in subject review

(This note has been prepared for use in reviewer training sessions. It gives guidance on making judgements, and should be read in conjunction with the section on making judgements in the QAA handbook.)

Guidance provided to reviewers during training

Judgements should be made in a way that will:

• be informative to students, employers, and institutions
• help providers identify readily any matters requiring remedial action, or strengths to be built upon
• provide a focus for any follow up that the Agency may have to carry out

This means, in particular, that where there are weaknesses or failures, these should be identified clearly. Judgements should not be made by averaging out strengths and weaknesses. Sometimes, that will mean that separate and additional judgements have to be made on certain programmes within the subject area that is being reviewed.

The standards judgement

This is a single, threshold judgement, of confidence or otherwise in standards, made after considering each of the following components:

• appropriateness of the intended learning outcomes (para 82)
• effectiveness of curriculum content and design, (para 83), including effectiveness in securing academic and intellectual progression (para 84)
• appropriateness of assessment in measuring the intended outcomes; security and integrity of the assessment process (para 85)
• actual student achievement in relation to the intended outcomes and the level of the award (para 86)

Unit of judgement

The largest unit of provision in respect of which a standards judgement may be made is the subject category, ie Law. If there are significant weaknesses present in some programmes, but not others, then separate, additional judgements should be made on those programmes that give rise to concern.

For example, reviewers may have confidence in the standards of honours degree programmes, but not in those of a taught masters programme. Similarly, reviewers may have confidence in the standards of a major part of the subject category, but not in another, for example confidence in Social Policy and Administration, but not in Social Work (para 87).

Identifying separately those programmes in which reviewers do not have confidence, or have only limited confidence, provides a sharp focus for remedial action by the provider, and follow-up action by the Agency.

Limited confidence

This is a judgement that is likely to be made rarely. It is not a half way house between ‘confidence’ and ‘no confidence’.

Before this judgement can be made, reviewers must have concluded that they have confidence in standards at present. It should only be made if there is an identifiable risk to standards in future. It might be appropriate if a major change to curriculum or assessment was planned, and reviewers had serious doubts about the effectiveness of the new arrangements.

Remember that the ‘confidence’ judgement can be made only if reviewers are satisfied both with current standards, and the prospect of those standards being maintained into the future. A ‘limited confidence’ judgement would be appropriate if the first condition was satisfied, but not the second (para 81).

No confidence

Reviewers must be satisfied with all of the matters listed in the bullet points above, if they are to make a ‘confidence’ judgement.

The first three deal with the design of the programmes, the fourth with the actual achievement of students. Reviewers cannot have confidence in the standards of provision if they are satisfied on only three out of the four matters. If they are not satisfied with one, then that will result in a judgement of ‘no confidence’.

The nature of the failure should be identified in the narrative, to assist remedial action and follow up. As noted above, if a failure is in respect of a single programme, or programmes at one level only, the ‘no confidence’ judgement may be restricted to the failing programmes, and a separate judgement made on the others.

The quality judgements

A series of judgements is made, on each of:

• teaching and learning (para 89)
• student progression (para 90)
• learning resources (para 91)

Unit of judgement

For each of the three aspects listed above, a single judgement may be made covering all of the provision under review. If more than one subject category is included in the review, it is still possible to make a single judgement.

Where subject categories have been aggregated by the provider, it is likely to be because of the way in which provision is offered to students, for example through a multi-disciplinary modular scheme. If the students experience a consistent quality of provision across all components of their study, it is reasonable for a single judgement to be made about the quality of the learning opportunities provided.

However, if weaknesses affect a single subject, or a particular level, or a separately identifiable group of programmes, then separate and additional judgements should be made.

The judgements

Most judgements will require reviewers to consider whether provision should be placed in the ‘commendable’ or the ‘approved, but&’ category (para 92).

The ‘approved, but…’ category should be used when there is identifiable weakness that should be addressed. The narrative of the report should identify clearly the nature of that weakness.

The ‘commendable’ category is likely to be appropriate in the majority of cases. The provision, in the aspect concerned, should be contributing “substantially to the achievement of the intended outcomes, with most elements demonstrating good practice”. This does not mean that it should be perfect, but neither should it have significant weaknesses.

If provision is less than adequate, it should be judged to be ‘failing’.

Exemplary features

Exemplary features will not be common. They relate to a specific feature of an aspect of provision. ‘Exemplary’ is not a fourth category of judgement, above the ‘commendable’ category. To be deemed ‘exemplary’ a feature must satisfy all three of the criteria listed in para 93.

The feature must:

• represent sector-leading best practice; and
• be worthy of dissemination to, and emulation by, other providers of comparable programmes; and
• make a significant contribution to the provision being assessed.

Excellence alone does not make a feature exemplary. For example, a unique programme might be of excellent quality, but its features would not be exemplary if no other programme existed that could emulate them. ‘Exemplariness’ is about innovation that can be used to promote enhancement elsewhere, it is not solely an accolade for the provider.

Review teams will need to draw upon their collective experience to judge whether a feature represents sector leading best practice. It will be a matter of both fact and judgement as to whether a feature makes a ‘significant contribution’ to the success of the provision, but incidental or marginal features should not be considered.

The critical test will be whether the feature is worthy of dissemination to, and emulation by other providers. Reviewers will wish to consider the practicality of a feature being emulated elsewhere. An exemplar is “a model for imitation” (Oxford shorter English dictionary). ‘Exemplary’ signifies not just excellence, but excellence which is generalisable and transferable.

Overall judgements of failure

A failing judgement in any aspect of quality, or a finding of no confidence in standards, will lead to the provision being regarded, overall, as failing. If the failing or no confidence judgement has been expressed in respect of the whole of the provision under review, then the whole of that provision fails, and the whole of it will be subject to further review within a year.

If the failing or no confidence judgement relates to a programme or group of programmes only, then it is those programmes that fail overall, and which will be subject to further review. In this case the narrative of the report will need to make clear precisely which parts of the provision are approved, and which parts are regarded as failing.

Maintenance and enhancements of standards and quality

Reviewers should not overlook the significance of the comments that they are asked to make by the QAA handbook (para 95). This does not call for a formalised judgement, of the sort made in relation to standards and the quality of learning opportunities.

Nevertheless, this is the part of the report in which reviewers are invited to express their confidence, or otherwise, in the overall ability of the provider to maintain and enhance quality and standards in the subject(s) under review.

In expressing their views, reviewers should bear in mind the matters that will be addressed in the institutional review component of subject review. (See part 2 of the QAA handbook.) Comments about the degree of confidence that reviewers have in institutional arrangements, in the light of their findings at subject level, are of particular value if they address issues concerning:

• programme approval, monitoring and review
• assessment of students
• external examining
• collaborative provision

If reviewers have doubts about the ability of the provider to maintain and enhance quality and standards, the cause of the doubt should be clearly articulated, so that it may be followed up in a subsequent institutional review. Similarly, strengths should be identified, as these may have a bearing on the intensity of scrutiny that may be required at institutional level.

Last Modified: 30 June 2010